[KYUUBI #7635][SPARK] Support Zstd compression for Arrow IPC query results - #7636
[KYUUBI #7635][SPARK] Support Zstd compression for Arrow IPC query results#7636byronwang123 wants to merge 8 commits into
Conversation
github-actions
Bot
added
kind:infra
…he engine jar Spark < 4.1 runtimes do not ship arrow-compression, so the compressed Arrow result path (and even the default none path when a LIMIT slices a batch, which always constructs VectorLoader with a decompression factory) would fail with NoClassDefFoundError unless users added jars manually. Bundle arrow-compression into the shaded engine jar, relocated under org.apache.kyuubi.shade.org.apache.arrow.compression, so it is self-contained on every supported Spark version and cannot conflict with the arrow-compression jar that Spark 4.1+ ships. The Arrow vector/memory/format modules and zstd-jni/commons-compress/ commons-io are excluded from shading: every supported Spark runtime provides them, and excluding them avoids classpath conflicts and native library relocation issues.
byronwang123
changed the title
|
the decision to choose "Arrow IPC compression" looks good to me, Arrow Java has implemented it since 8.0.0, and Spark 3.5 ships Arrow 12.0.1, so that covers all supported Spark versions. cc @cfmcgrady, could you take a look since you are the author of this feature? |
…set#toArrowBatchRdd The arrow-compression library is now relocated into the engine jar, so the reflective fallback that invoked the vanilla Spark Dataset#toArrowBatchRdd is no longer needed; use the plan-based helper directly so the Kyuubi zstd codec applies. Also reject lz4 explicitly instead of silently falling back to no compression, and add server-side round-trip coverage.
…istic and fail fast on unsupported codec
…arrow-compression dependency chain
… on the engine side and exclude zstd-jni from the shaded client
…lity at runtime before using zstd
|
code change looks almost good to me, and I need to check the packaging result and verify it on a real setup before approval (hope I can find time to finish it in one or two days) |
|
And seems you deleted this part from the PR description? But this is a legally mandated item.
|
98ed9ec to
34e8ac6
Compare
Thanks for taking another look. I've addressed the remaining naming comment and updated the PR description to reflect the final runtime/packaging behavior, including the generative AI disclosure. Appreciate you checking the packaging and real setup. |
|
CI failure is related |
2 participants
Why are the changes needed?
Kyuubi's Arrow result format reduces serialization/deserialization overhead, but Arrow IPC results are currently transferred without compression. For large result sets, network transfer can become a significant part of fetch latency.
This follows up on #3877 (related issue #3865). Spark 4.1 introduced the upstream Arrow compression configurations (
SPARK-54134), so this PR follows Spark's configuration and Arrow IPC compression semantics instead of introducing a Kyuubi-specific protocol.The main changes are:
none(default) andzstdfor Arrow IPC query results, including ZSTD compression-level configuration.LIMITcrosses a batch boundary.arrow-compressionoptional on the Spark engine side. The defaultnonepath requires no additional dependency; when zstd is enabled, Kyuubi checks the required Arrow compression capability at runtime and reports a clear error if it is unavailable.kyuubi-hive-jdbc-shadedto include/relocatearrow-compression, while excludingzstd-jniand declaring it optional.commons-compress, since LZ4 is not supported.For Spark runtimes that do not provide a compatible
arrow-compressionimplementation, users enabling zstd need to add one to the Spark classpath. The defaultnonebehavior remains unchanged.A cluster benchmark with Kyuubi 1.13.0-SNAPSHOT + Spark 3.5.9 showed 78.8%-81.8% fewer Arrow IPC bytes, with fetch time reduced from 19.3s to 7.5s in the 8-column case and from 205.8s to 94.1s in the 34-column case.
How was this patch tested?
Added/updated coverage for:
LIMITboundaries.nonepath with noarrow-compressionon the engine classpath.none -> zstd -> noneswitching within the same engine.Targeted engine tests and
KyuubiOperationPerUserSuitepass on JDK 17 / Spark 3.5.8. The shaded artifacts were also checked to verify the intendedarrow-compression/zstd-jnipackaging.Was this patch authored or co-authored using generative AI tooling?
Assisted-by: DeepSeek Harness:deepseek-v4-pro